System and method for local three dimensional volume reconstruction using a standard fluoroscope

ABSTRACT

A system and method for constructing fluoroscopic-based three dimensional volumetric data from two dimensional fluoroscopic images including a computing device configured to facilitate navigation of a medical device to a target area within a patient and a fluoroscopic imaging device configured to acquire a fluoroscopic video of the target area about a plurality of angles relative to the target area. The computing device is configured to determine a pose of the fluoroscopic imaging device for each frame of the fluoroscopic video and to construct fluoroscopic-based three dimensional volumetric data of the target area in which soft tissue objects are visible using a fast iterative three dimensional construction algorithm.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application Ser.No. 62/201,750, filed on Aug. 6, 2015, the entire contents of which areincorporated by reference herein.

BACKGROUND

Technical Field

The present disclosure relates to a system, apparatus, and method ofnavigation and position confirmation for surgical procedures. Moreparticularly, the present disclosure relates to a system and method forconstructing a fluoroscopic-based three dimensional volume from twodimensional fluoroscopic images captured using a standard fluoroscopicimaging device.

Description of Related Art

There are several commonly applied methods for treating various maladiesaffecting organs including the liver, brain, heart, lung and kidney.Often, one or more imaging modalities, such as magnetic resonanceimaging, ultrasound imaging, computed tomography (CT), as well as othersare employed by clinicians to identify areas of interest within apatient and ultimately targets for treatment.

An endoscopic approach has proven useful in navigating to areas ofinterest within a patient, and particularly so for areas within luminalnetworks of the body such as the lungs. To enable the endoscopic, andmore particularly the bronchoscopic, approach in the lungs,endobronchial navigation systems have been developed that use previouslyacquired MRI data or CT image data to generate a three dimensionalrendering or volume of the particular body part such as the lungs. Inparticular, previously acquired images, acquired from an MRI scan or CTscan of the patient, are utilized to generate a three dimensional orvolumetric rendering of the patient.

The resulting volume generated from the MRI scan or CT scan is thenutilized to create a navigation plan to facilitate the advancement of anavigation catheter (or other suitable device) through a bronchoscopeand a branch of the bronchus of a patient to an area of interest.Electromagnetic tracking may be utilized in conjunction with the CT datato facilitate guidance of the navigation catheter through the branch ofthe bronchus to the area of interest. In certain instances, thenavigation catheter may be positioned within one of the airways of thebranched luminal networks adjacent to, or within, the area of interestto provide access for one or more medical instruments.

Thus, in order to generate a navigation plan, or in order to evengenerate a three dimensional or volumetric rendering of the patient'sanatomy, such as the lung, a clinician is required to utilize an MRIsystem or CT system to acquire the necessary image data for constructionof the three dimensional volume. An MRI system or CT-based imagingsystem is extremely costly, and in many cases not available in the samelocation as the location where a navigation plan is generated or where anavigation procedure is carried out.

A fluoroscopic imaging device is commonly located in the operating roomduring navigation procedures. The standard fluoroscopic imaging devicemay be used by a clinician to visualize and confirm the placement of atool after it has been navigated to a desired location. However,although standard fluoroscopic images display highly dense objects suchas metal tools and bones as well as large soft-tissue objects such asthe heart, the fluoroscopic images have difficulty resolving smallsoft-tissue objects of interest such as lesions. Further, thefluoroscope image is only a two dimensional projection. In order to beable to see small soft-tissue objects in three dimensional space, anX-ray volumetric reconstruction is needed. Several solutions exist thatprovide three dimensional volume reconstruction of soft-tissues such asCT and Cone-beam CT which are extensively used in the medical world.These machines algorithmically combine multiple X-ray projections fromknown, calibrated X-ray source positions into three dimensional volumein which the soft-tissues are visible.

In order to navigate tools to a remote soft-tissue target for biopsy ortreatment, both the tool and the target should be visible in some sortof a three dimensional guidance system. The majority of these systemsuse some X-ray device to see through the body. For example, a CT machinecan be used with iterative scans during procedure to provide guidancethrough the body until the tools reach the target. This is a tediousprocedure as it requires several full CT scans, a dedicated CT room andblind navigation between scans. In addition, each scan requires thestaff to leave the room. Another option is a Cone-beam CT machine whichis available in some operation rooms and is somewhat easier to operate,but is expensive and like the CT only provides blind navigation betweenscans, requires multiple iterations for navigation and requires thestaff to leave the room.

Accordingly, there is a need for a system that can achieve the benefitsof the CT and Cone-beam CT three dimensional image guidance without theunderlying costs, preparation requirements, and radiation side effectsassociated with these systems.

SUMMARY

The present disclosure is directed to a system and method forconstructing three dimensional volumetric data in which smallsoft-tissue objects are visible from a video stream (or plurality ofimages) captured by a standard fluoroscopic imaging device available inmost procedure rooms. The fluoroscopic-based constructed threedimensional volumetric data may be used for guidance, navigationplanning, improved navigation accuracy, navigation confirmation, andtreatment confirmation.

Soft tissue objects are not visible in standard fluoroscopic images andvideo because they are obscured by dense objects such as dense tissue.The present disclosure is directed to a system and method for creating athree-dimensional pseudo-volume in which the obscuring objects arefiltered based on the three dimensional position, and then projectedback into two dimensions. In one aspect, multiple fluoroscopic images,each captured at different angles, are utilized to construct the threedimensional pseudo-volume data. The present disclosure describes asystem and method which is capable of constructing the three dimensionalpseudo-volume data utilizing fluoroscopic images captured from a shortrange of angles relative to a patient or target region of a patient.Additionally, the present disclosure is also directed to a system andmethod for improving a previously created three dimensional renderingutilizing two dimensional images, or video, captured by a standardfluoroscopic imaging device.

As described in greater detail below, one aspect of the presentdisclosure is to determine three dimensional positions of features inthe fluoroscopic video, such as three dimensional catheter position,three dimensional target tissue (for example a lesion) position, etc. Inorder to accomplish this, the pose of the fluoroscopic imaging devicefor each frame must be determined or known. If an external anglemeasurement device coupled to the fluoroscopic imaging device isutilized, then the angles and pose of the imaging device is known foreach frame. However, when external measurement devices are not utilizedother techniques are employed to determine the poses of the fluoroscopicimaging device for each frame, as described in greater detail below. Forexample, previously acquired CT scan data may be registered to thefluoroscopic video in order to algorithmically find the pose of thefluoroscope for each frame of the captured video. Alternatively, bytracking a few visible markers (two dimensional visible features) in thefluoroscopic video, the pose and three dimensional positions may besolved together by using some structure from motion technique. In someaspects, easier techniques in which the poses are known (by an anglemeasurement device) may be utilized and only the three dimensionalfeatures' positions need to be solved. In order to correct formovements, at least one marker (or a surgical device such as a cathetertip) may be utilized.

Multiple fluoroscope two dimensional images can be processedalgorithmically to create pseudo three dimensional volumetric data,similar to Cone-beam CT, but with varying arbitrary angle range andfluoroscope poses, where the fluoroscope is rotated manually. The anglerange may be very small (˜30°) which may result in poor threedimensional reconstruction quality. The algorithm used is an iterativeaccelerated projection/back-projection method which unlike the analyticalgorithms (Radon transform, FDK . . . ) doesn't assume anypredetermined angle range or angle rotation rate. In order to overcomethe poor three dimensional reconstruction, instead of displaying the rawthree dimensional reconstructed data to the user, the three dimensionalreconstruction may be cropped around the area of interest (also referredto herein as the “FluoroCT Blob”). The cropped three dimensional data isthen reprojected into two dimensional virtual fluoroscope images inwhich local soft-tissue features are visible. The reprojected twodimensional images are of good quality (compared to the poor threedimensional reconstruction), especially if the projections are done fromthe same fluoroscope poses as were seen in the video.

In order to reconstruct the three dimensional data, the pose of thefluoroscopic imaging device must be determined for each two dimensionalfluoroscope frame in the video, relative to some fixed coordinatesystem. The pose of the fluoroscopic imaging device for each framecaptured may be determined using any of the methods described below. Forexample the pose may be determined using an external measurement devicecoupled to the fluoroscopic imaging device.

Additionally, or alternatively, the pose may be determined using anexternal measurement device and a single marker or a catheter tip.Specifically, in some cases, the fluoroscope (or C-arm) may jitterduring rotation, in which case some stabilization is needed.Unfortunately, due to filtering, the angle measurement device may stillreport smooth angles throughout the video, ignoring the high-frequencieswhich are present in the actual camera poses. In this case, a single twodimensional marker can be tracked throughout the video and used tostabilize the camera poses, or to increase their accuracy at the area ofthe marker. Since the marker will usually be located at the region ofinterest, this increases the camera pose accuracy in this region, andthus improves the three dimensional reconstruction quality. This methodcan also be used to compensate for patient body movement such asbreathing during the video. Instead of using the camera pose as reportedby the angle measurement device, a compensated camera pose is computedusing the tracked two dimensional marker, such that all camera poseswill be correct relative to it. The single marker used can be the tip ofa tool or a catheter which is currently inserted to the patient.

Additionally, or alternatively, the pose may be determined viaregistration of the fluoroscopic video to previously acquired CT data.Specifically, a previously acquired CT of the patient may be available.In this case, each frame of the fluoroscope video can be registered to avirtual fluoroscopic frame of the CT (camera pose is searched in CTspace until the virtual fluoroscopic image, corresponding to the camerapose, matches the one seen in the video). In this way, the camera poseis realized using image-based features matching.

Additionally, or alternatively, the camera pose may be determined usingstructure from motion techniques. Specifically, if numerous twodimensional features can be tracked throughout the two dimensional videoframes, from beginning to end, then these two dimensional features canbe used to realize camera poses (together with three dimensional featurepositions) for each frame. These features can be artificial markerswhich were introduced to the patient during procedure.

Filtering the obscuring tissue from the tissue of interest can be doneby cropping at a distance from the center of the generated threedimensional pseudo-volume data, cropping at a distance from thetool/catheter, or registering to previously obtained three dimensionalvolume data (CT) and using it to know which objects to filter.

Once the soft-tissue (such as a target lesion) is visible in the threedimensional reconstructed data (or in the two dimensional enhancedFluoroCT images), all three dimensional information in fluoroscopecoordinates can be obtained. When working with the raw three dimensionalreconstructed data the three dimensional information is obtaineddirectly from the data. Alternatively, when working with the twodimensional enhanced FluoroCT images, 2-angles markings are needed inorder to realize three dimensional positions (triangulation).

The obtained three dimensional data may be utilized for confirmation oftool to soft-tissue target three dimensional relation. For example, thedata may be used to determine whether the tool reached the target, theorientation of the tool relative to the target, the distance between thetool and the target, or whether the target falls within an ablation zoneof the tool.

Additionally, or alternatively, the three dimensional data may beutilized for correction of navigation. For example, the threedimensional positions can be transformed from fluoroscope coordinatesinto navigation system coordinates to improve navigation system accuracyat the region of interest. In one aspect, when utilized in an EMNsystem, fluoroscope coordinates can be transformed to antennacoordinates by assuming that the C-arm is perfectly perpendicular to theantenna and matching the catheter tip, seen in the fluoroscope video, tothe catheter position in antenna at the time the video was taken.Fluoroscopic to/from antenna registration can also be achieved bycomputing the angle between the C-arm and the antenna using the Earth'smagnetic field, attaching an EMN sensor to the fluoroscopic imagingdevice, or aligning a known two dimensional feature of the antenna.

Aspects of the present disclosure are described in detail with referenceto the figures wherein like reference numerals identify similar oridentical elements. As used herein, the term “distal” refers to theportion that is being described which is further from a user, while theterm “proximal” refers to the portion that is being described which iscloser to a user.

According to one aspect of the present disclosure, a system forconstructing fluoroscopic-based three dimensional volumetric data fromtwo dimensional fluoroscopic images is provided. The system includes acomputing device configured to facilitate navigation of a medical deviceto a target area within a patient and a fluoroscopic imaging deviceconfigured to acquire a fluoroscopic video of the target area about aplurality of angles relative to the target area. The computing device isconfigured to determine a pose of the fluoroscopic imaging device foreach frame of the fluoroscopic video and to construct fluoroscopic-basedthree dimensional volumetric data of the target area in which softtissue objects are visible using a fast iterative three dimensionalconstruction algorithm. In aspects, the system is configured todetermine the pose of the fluoroscopic imaging device by knowing theangle range of movement of the device and computing the relativerotational speed of the device along the range.

The medical device may be a catheter assembly including an extendedworking channel configured to be positioned within a luminal network ofthe patient or the medical device may be a radio-opaque markerconfigured to be placed within the target area. The radio-opaque markeris at least partially visible in the fluoroscopic video acquired.

The computing device may be further configured to create virtualfluoroscopic images of the patient from previously acquired CTvolumetric data and register the generated virtual fluoroscopic imageswith the acquired fluoroscopic video. The pose of the fluoroscopicimaging device for each frame of the fluoroscopic video may bedetermined based on the registration between the fluoroscopic video andthe virtual fluoroscopic images.

The computing device may further be configured to detect frames missingfrom the fluoroscopic video and supplement the detected missing frameswith corresponding virtual fluoroscopic images. The fluoroscopic-basedthree dimensional volumetric data may be constructed based on thefluoroscopic video and the corresponding virtual fluoroscopic images. Inone aspect, the fluoroscopic-based three dimensional volumetric data maybe registered with previously acquired CT data using image-basedtechniques such as “mutual information.” The fluoroscopic-based threedimensional volumetric data may be registered globally to the previouslyacquired CT data or locally, at the proximity of the target area ofinterest. A deep-learning based approach may be utilized in which thecomputing device “sees” many examples of suitable and non-suitableregistrations and learns how to register the very two differentmodalities.

Additionally, the computing device may be further configured to trackthe two dimensional position or orientation of the medical devicenavigated to the target region throughout the fluoroscopic video. Thecomputing device may be further configured to reconstruct positions ofthe medical device throughout the fluoroscopic video using astructure-from-motion technique. The pose of the fluoroscopic imagingdevice for each frame of the fluoroscopic video may be determined basedon the reconstructed positions. Additionally, or alternatively, the poseof the fluoroscopic imaging device for each frame of the fluoroscopicvideo may be determined based on an external angle measuring device. Theexternal angle measuring device may include an accelerometer, agyroscope, or a magnetic field sensor coupled to the fluoroscopicimaging device. Additionally, in aspects, the computing device may beconfigured to synchronize the captured frames of the target area andcompensate for shifts in the fluoroscopic imaging device or patientmovement to correct construction of the fluoroscopic-based threedimensional volumetric data. Additionally, or alternatively, thecomputing device may be configured to crop a region of interest from thefluoroscopic-based three dimensional volumetric data, project thecropped region of interest onto the captured frames, and sharpen orintensify at least one of the region of interest or the captured frameto identify soft tissue objects.

In yet another aspect of the present disclosure a method forconstructing fluoroscopic-based three dimensional volumetric data fromtwo dimensional fluoroscopic images is provided. The method includesnavigating a medical device to a target area within a patient, acquiringa fluoroscopic video of the target area about a plurality of anglesrelative to the target area using a fluoroscopic imaging device,determining a pose of the fluoroscopic imaging device for each frame ofthe fluoroscopic video, and constructing fluoroscopic-based threedimensional volumetric data of the target area in which soft tissueobjects are visible using a fast iterative three dimensionalconstruction algorithm. The medical device may be a catheter assemblyincluding an extended working channel configured to be positioned withina luminal network of the patient or the medical device may be aradio-opaque marker configured to be placed within the target area. Theradio-opaque marker is at least partially visible in the fluoroscopicvideo acquired.

The method may further include creating virtual fluoroscopic images ofthe patient from previously acquired CT volumetric data, and registeringthe fluoroscopic video with the virtual fluoroscopic images, whereindetermining the pose of the fluoroscopic imaging device for each frameof the fluoroscopic video is based on the registration between thefluoroscopic video and the virtual fluoroscopic images. The method mayfurther include detecting frames missing from the fluoroscopic video andsupplementing the detected missing frames with corresponding virtualfluoroscopic images. Additionally, in aspects of the disclosure, themethod may further include tracking the two dimensional position ororientation of the medical device navigated to the target regionthroughout the fluoroscopic video.

The positions of the medical device throughout the fluoroscopic videomay be reconstructed using a structure-from-motion technique. The poseof the fluoroscopic imaging device for each frame of the fluoroscopicvideo may be determined based on the reconstructed positions.Additionally, or alternatively, the pose of the fluoroscopic imagingdevice for each frame of the fluoroscopic video may be determined basedon an external angle measuring device. The external angle measuringdevice may include an accelerometer, a gyroscope, or a magnetic fieldsensor coupled to the fluoroscopic imaging device. Additionally, inaspects, the method may include synchronizing the captured frames of thetarget area and compensating for shifts in the fluoroscopic imagingdevice or patient movement to correct construction of thefluoroscopic-based three dimensional volumetric data. Additionally, oralternatively, the method may include cropping a region of interest fromthe fluoroscopic-based three dimensional volumetric data, projecting thecropped region of interest onto the captured frames, and sharpening orintensifying at least one of the region of interest or the capturedframe to identify soft tissue objects.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects and embodiments of the present disclosure are describedhereinbelow with references to the drawings, wherein:

FIG. 1 is a perspective view of one illustrative embodiment of anelectromagnetic navigation (EMN) system incorporating a fluoroscopicimaging device in accordance with the present disclosure;

FIG. 2 illustrates a fluoroscopic imaging device model;

FIG. 3A is a flow chart of a method for constructing a three dimensionalvolume using a plurality of radio-opaque markers;

FIG. 3B is an illustration of an example of frames of a fluoroscopicvideo captured by a fluoroscopic imaging device showing markers and anextended working channel of a catheter assembly positioned within atarget region of a patient in accordance with the instant disclosure;

FIG. 4 is a flow chart of a method for constructing a three dimensionalvolume using either a single radio-opaque marker or the tip of anextended working channel of a catheter assembly;

FIG. 5 is a flow chart of a method for constructing a three dimensionalvolume using either a single radio-opaque marker or the tip of anextended working channel of a catheter assembly in conjunction with anangle measurement device;

FIG. 6 is a flow chart of a method for three dimensional modelconstruction in accordance with the instant disclosure;

FIG. 7 is an illustration of a frame of an original video captured by afluoroscopic imaging device, an image of the frame after being rampfiltered, and the resulting three dimensional volume; and

FIG. 8 is an illustration of a three dimensional construction generatedaccording to a given angle range of fluoroscopic images/video.

DETAILED DESCRIPTION

The present disclosure is directed to a system and method forconstructing local three dimensional volumetric data, in which smallsoft-tissue objects are visible, from a video stream captured by astandard fluoroscopic imaging device available in most procedure rooms.The constructed fluoroscopic-based local three dimensional volumetricdata may be used for guidance, navigation planning, improved navigationaccuracy, navigation confirmation, and treatment confirmation.

FIG. 1 depicts an Electromagnetic Navigation (EMN) system 100 configuredfor reviewing CT image data to identify one or more targets, planning apathway to an identified target (planning phase), navigating an extendedworking channel (EWC) 12 of a catheter assembly to a target (navigationphase) via a user interface, and confirming placement of the EWC 12relative to the target. One such EMN system is the ELECTROMAGNETICNAVIGATION BRONCHOSCOPY® system currently sold by Medtronic PLC. Thetarget may be tissue of interest identified by review of the CT imagedata during the planning phase. Following navigation, a medicalinstrument, such as a biopsy tool or other tool, may be inserted intothe EWC 12 to obtain a tissue sample from the tissue located at, orproximate to, the target.

As shown in FIG. 1, EWC 12 is part of a catheter guide assembly 40. Inpractice, the EWC 12 is inserted into bronchoscope 30 for access to aluminal network of the patient “P.” Specifically, EWC 12 of catheterguide assembly 40 may be inserted into a working channel of bronchoscope30 for navigation through a patient's luminal network. A locatable guide(LG) 32, including a sensor 44 is inserted into the EWC 12 and lockedinto position such that the sensor 44 extends a desired distance beyondthe distal tip of the EWC 12. The position and orientation of the sensor44 relative to the reference coordinate system, and thus the distalportion of the EWC 12, within an electromagnetic field can be derived.Catheter guide assemblies 40 are currently marketed and sold byMedtronic PLC under the brand names SUPERDIMENSION® Procedure Kits, orEDGE™ Procedure Kits, and are contemplated as useable with the presentdisclosure. For a more detailed description of the catheter guideassemblies 40, reference is made to commonly-owned U.S. PatentPublication No. 2014/0046315, filed on Mar. 15, 2013, by Ladtkow et al,U.S. Pat. Nos. 7,233,820, and 9,044,254, the entire contents of each ofwhich are hereby incorporated by reference.

EMN system 100 generally includes an operating table 20 configured tosupport a patient “P,” a bronchoscope 30 configured for insertionthrough the patient's “P's” mouth into the patient's “P's” airways;monitoring equipment 120 coupled to bronchoscope 30 (e.g., a videodisplay, for displaying the video images received from the video imagingsystem of bronchoscope 30); a tracking system 50 including a trackingmodule 52, a plurality of reference sensors 54 and a transmitter mat 56;and a computing device 125 including software and/or hardware used tofacilitate identification of a target, pathway planning to the target,navigation of a medical instrument to the target, and confirmation ofplacement of an EWC 12, or a suitable device therethrough, relative tothe target.

A fluoroscopic imaging device 110 capable of acquiring fluoroscopic orx-ray images or video of the patient “P” is also included in thisparticular aspect of system 100. The images, series of images, or videocaptured by the fluoroscopic imaging device 110 may be stored within thefluoroscopic imaging device 110 or transmitted to computing device 125for storage, processing, and display. Additionally, the fluoroscopicimaging device 110 may move relative to the patient “P” so that imagesmay be acquired from different angles or perspectives relative to thepatient “P” to create a fluoroscopic video. In one aspect of the presentdisclosure, fluoroscopic imaging device 110 includes an anglemeasurement device 111 which is configured to measure the angle of thefluoroscopic imaging device 110 relative to the patient “P.” Anglemeasurement device 111 may be an accelerometer. Fluoroscopic imagingdevice 110 may include a single imaging device or more than one imagingdevice. In embodiments including multiple imaging devices, each imagingdevice may be a different type of imaging device or the same type.Further details regarding the imaging device 110 are described in U.S.Pat. No. 8,565,858, which is incorporated by reference in its entiretyherein.

Computing device 125 may be any suitable computing device including aprocessor and storage medium, wherein the processor is capable ofexecuting instructions stored on the storage medium. The computingdevice 125 may further include a database configured to store patientdata, CT data sets including CT images, fluoroscopic data sets includingfluoroscopic images and video, navigation plans, and any other suchdata. Although not explicitly illustrated, the computing device 125 mayinclude inputs, or may otherwise be configured to receive, CT data sets,fluoroscopic images/video and other data described herein. Additionally,computing device 125 includes a display configured to display graphicaluser interfaces. Computing device 125 may be connected to one or morenetworks through which one or more databases may be accessed.

With respect to the planning phase, computing device 125 utilizespreviously acquired CT image data for generating and viewing a threedimensional model of the patient's “P's” airways, enables theidentification of a target on the three dimensional model(automatically, semi-automatically, or manually), and allows fordetermining a pathway through the patient's “P's” airways to tissuelocated at and around the target. More specifically, CT images acquiredfrom previous CT scans are processed and assembled into a threedimensional CT volume, which is then utilized to generate a threedimensional model of the patient's “P's” airways. The three dimensionalmodel may be displayed on a display associated with computing device125, or in any other suitable fashion. Using computing device 125,various views of the three dimensional model or enhanced two dimensionalimages generated from the three dimensional model are presented. Theenhanced two dimensional images may possess some three dimensionalcapabilities because they are generated from three dimensional data. Thethree dimensional model may be manipulated to facilitate identificationof target on the three dimensional model or two dimensional images, andselection of a suitable pathway through the patient's “P's” airways toaccess tissue located at the target can be made. Once selected, thepathway plan, three dimensional model, and images derived therefrom, canbe saved and exported to a navigation system for use during thenavigation phase(s). One such planning software is the ILOGIC® planningsuite currently sold by Medtronic PLC.

With respect to the navigation phase, a six degrees-of-freedomelectromagnetic tracking system 50, e.g., similar to those disclosed inU.S. Pat. Nos. 8,467,589, 6,188,355, and published PCT Application Nos.WO 00/10456 and WO 01/67035, the entire contents of each of which areincorporated herein by reference, or other suitable positioningmeasuring system, is utilized for performing registration of the imagesand the pathway for navigation, although other configurations are alsocontemplated. Tracking system 50 includes a tracking module 52, aplurality of reference sensors 54, and a transmitter mat 56. Trackingsystem 50 is configured for use with a locatable guide 32 andparticularly sensor 44. As described above, locatable guide 32 andsensor 44 are configured for insertion through an EWC 12 into apatient's “P's” airways (either with or without bronchoscope 30) and areselectively lockable relative to one another via a locking mechanism.

Transmitter mat 56 is positioned beneath patient “P.” Transmitter mat 56generates an electromagnetic field around at least a portion of thepatient “P” within which the position of a plurality of referencesensors 54 and the sensor element 44 can be determined with use of atracking module 52. One or more of reference sensors 54 are attached tothe chest of the patient “P.” The six degrees of freedom coordinates ofreference sensors 54 are sent to computing device 125 (which includesthe appropriate software) where they are used to calculate a patientcoordinate frame of reference. Registration, as detailed below, isgenerally performed to coordinate locations of the three dimensionalmodel and two dimensional images from the planning phase with thepatient's “P's” airways as observed through the bronchoscope 30, andallow for the navigation phase to be undertaken with precise knowledgeof the location of the sensor 44, even in portions of the airway wherethe bronchoscope 30 cannot reach. Further details of such a registrationtechnique and their implementation in luminal navigation can be found inU.S. Patent Application Pub. No. 2011/0085720, the entire content ofwhich is incorporated herein by reference, although other suitabletechniques are also contemplated.

Registration of the patient's “P's” location on the transmitter mat 56is performed by moving LG 32 through the airways of the patient's “P.”More specifically, data pertaining to locations of sensor 44, whilelocatable guide 32 is moving through the airways, is recorded usingtransmitter mat 56, reference sensors 54, and tracking module 52. Ashape resulting from this location data is compared to an interiorgeometry of passages of the three dimensional model generated in theplanning phase, and a location correlation between the shape and thethree dimensional model based on the comparison is determined, e.g.,utilizing the software on computing device 125. In addition, thesoftware identifies non-tissue space (e.g., air filled cavities) in thethree dimensional model. The software aligns, or registers, an imagerepresenting a location of sensor 44 with a the three dimensional modeland two dimensional images generated from the three dimension model,which are based on the recorded location data and an assumption thatlocatable guide 32 remains located in non-tissue space in the patient's“P's” airways. Alternatively, a manual registration technique may beemployed by navigating the bronchoscope 30 with the sensor 44 topre-specified locations in the lungs of the patient “P”, and manuallycorrelating the images from the bronchoscope to the model data of thethree dimensional model.

Following registration of the patient “P” to the image data and pathwayplan, a user interface is displayed in the navigation software whichsets for the pathway that the clinician is to follow to reach thetarget. One such navigation software is the ILOGIC® navigation suitecurrently sold by Medtronic PLC.

Once EWC 12 has been successfully navigated proximate the target asdepicted on the user interface, the locatable guide 32 may be unlockedfrom EWC 12 and removed, leaving EWC 12 in place as a guide channel forguiding medical instruments including without limitation, opticalsystems, ultrasound probes, marker placement tools, biopsy tools,ablation tools (i.e., microwave ablation devices), laser probes,cryogenic probes, sensor probes, and aspirating needles to the target.

Having described the components of system 100 depicted in FIG. 1, thefollowing description of FIGS. 2-8 provides an exemplary workflow ofusing the components of system 100, including the fluoroscopic imagingdevice 110, to construct local three dimensional volumetric data of adesired region of interest using the fluoroscopic imaging device 110 ofsystem 100. The systems and methods described herein may be useful forvisualizing a particular target region of a patient utilizing imagingdevices which are commonly located within a surgical setting during EMNprocedures, thereby obviating the need for subsequent MRI or CT scans.

Turning now to FIG. 2, a fluoroscopic imaging device 110 model isillustrated. The fluoroscopic imaging device 110 includes an X-raysource 201 and a detector 203. The detector 203 defines a plurality oftwo dimensional pixels p. Each two dimensional pixel p_(i) is associatedwith a single X-ray beam l_(i), traversing three dimensional space fromthe X-ray source 201 to the detector 203. The detector 203 size D andthe source-detector distance SDD determine the Field-of-View angle usingthe following formula:

$\theta = {2{{\tan^{- 1}\left( \frac{D}{2{SDD}} \right)}.}}$

Different fluoroscopes differ primarily by detector size andsource-detector distance. In the fluoroscopic data, pixels are notnormalized to a specific scale. Their brightness depends on thegain/exposure used by the fluoroscope and on other objects in the scenewhich the X-rays traverse between the source and the detector such asthe table, the surgical device, etc. In the CT data, each voxel ismeasured in Hounsfield units. Hounsfield units are measured with respectto the brightness observed of water and air using the following formula:

${HU} = {1000{\frac{\mu - \mu_{water}}{\mu_{water} - \mu_{air}}.}}$

μ are attenuation coefficients. They range from 0 to infinity andmeasure how difficult it is for an X-ray beam l_(i) to traverse throughmatter of the three dimensional volume 205. “Thicker” matter has alarger μ. HU scales attenuation coefficients such that air is placed at−1000, water at 0 and thicker matter goes up to infinity.

Using the Beer-Lambert law, each two dimensional pixel p_(i) in thefluoroscope's detector 203 is given by:

p_(i, j) = I₀exp (−∫_(l_(i, j))μ(l)d l).

μ(x, y, z) is the attenuation coefficient of the three dimensionalvolume 205 at position (x, y, z). I₀ is the X-ray source 201 energy(higher energy produces brighter image). For simplicity, assume I₀=1.Taking the logarithm, each two dimensional pixel p_(i) is representedby:

log (p_(i, j)) = −∫_(l_(i, j))μ(l)d l.

This is true for each two dimensional detector pixel p_(i) whichprovides a linear equation on the three dimensional attenuationcoefficients. If many such equations are utilized (to solve for each twodimensional pixel p_(i) of the detector 203) then the attenuationcoefficients may be solved and the three dimensional volume data of thethree dimensional volume 205 can be constructed.

Fluoroscope to CT—Discretization:

The three dimensional volume 205 can be divided into a discrete grid,with voxels sitting at (x_(k), y_(k), z_(k)). Thus, the equation forsolving each two dimensional pixel p_(i) can then be written as:

${\log\left( p_{i,j} \right)} = {- {\sum\limits_{k}{w_{k}^{i,j}{{\mu\left( {x_{k},y_{k},z_{k}} \right)}.}}}}$

The left hand side of the equation above is the observed two dimensionalpixel p_(i) of the detector 203 and the right hand side of the equationabove is a weighted sum of the attenuation coefficients (that is to besolved) with w_(k) ^(i,j) which are determined by the known fluoroscopicimaging device position of the X-ray source 201.

In order to be able to solve for the volumetric attenuation coefficientvalues enough linear equations are needed, that is, enough twodimensional observations of different two dimensional pixels p arerequired. Standard detectors usually have 1024×1024 pixels, where eachpixel is represented by a single equation. Therefore, many twodimensional observations are required to be solved for (many twodimensional observed pixels) to reconstruct a three dimensional volume(to solve for many voxels). That is many two dimensional pixels arerequired to solve for the many voxels. In order for the weights in theequations to be known, the fluoroscopic imaging device X-ray source 201configuration (position, orientation, field of view) must be known.

In use, the three dimensional volume 205 is a portion of a patient'sbody. A fluoroscopic video which is comprised of multiple fluoroscopeimages (as frames of the video) are taken from many different locationsrelative to the patient, for example a 180° rotation about the patient,to acquire multiple observations of two dimensional pixels p atdifferent positions relative to the patient. As will be described ingreater detail below, the position of the fluoroscopic imaging devicerelative to the patient at a given time can be determined using multipletechniques, including structure-from-motion analysis of radio-opaquemarkers placed within the patient (FIGS. 3A-3B), a registration betweenthe captured fluoroscopic images/video and generated virtualfluoroscopic images (FIG. 4), or by an external angle measurement devicesuch as an accelerometer, gyroscope, or magnetic field sensor (FIG. 5).In one aspect, the system may correct for patient movements whenmeasuring angles when a marker is utilized which moves with the movementof the body. Specifically, all of the two dimensional fluoroscopicimages may be synched to the same three dimensional position based onthe position of the placed marker in each of the images.

Turning now to FIGS. 3A-3B and 4-6, methods for constructing a localthree dimensional volume of a target region using a standardfluoroscopic imaging device (such as the fluoroscopic imaging device 110of FIG. 1) in conjunction with a system such as the system described inFIG. 1 will now be described with particular detail. Although themethods illustrated and described herein are illustrated and describedas being in a particular order and requiring particular steps, any ofthe methods may include some or all of the steps and may be implementedin any order not specifically described.

The fluoroscopic-based three dimensional volume generated by any of themethods described below can be incorporated into system 100 for multiplepurposes. For example, the fluoroscopic-based three dimensional volumecan be registered with the previously generated three dimensionalvolumetric data that was utilized for navigation of the medicalinstrument. The system 100 may utilize the registration between thefluoroscopic-based three dimensional volume and the previously acquiredthree dimensional volumetric data to update the calculated position ofthe sensor 44 (FIG. 1) which has been placed in the body of the patient.More particularly, the three dimensional model of a patient's lungs,generated from previously acquired CT scans, may not provide a basissufficient for accurate guiding of medical instruments to a targetduring an electromagnetic navigation procedure. In certain instances,the inaccuracy is caused by deformation of the patient's lungs duringthe procedure relative to the lungs at the time of the acquisition ofthe previously acquired CT data. This deformation (CT-to-Bodydivergence) may be caused by many different factors, for example:sedation vs. no sedation, bronchoscope changing patient pose and alsopushing the tissue, different lung volume because CT was in inhale whilenavigation is during breathing, different bed, day, etc. Thus, anotherimaging modality is necessary to visualize targets and/or a terminalbronchial branch, and enhance the electromagnetic navigation procedureby correcting the navigation during the procedure, enablingvisualization of the target, and confirming placement of the surgicaldevice during the procedure. For this purpose, the system describedherein processes and converts image data captured by the fluoroscopicimaging device 110, as will be described in detail below. Thisfluoroscopic image data may be utilized to identify such targets andterminal bronchial branches or be incorporated into, and used to update,the data from the CT scans in an effort to provide a moreaccurate/correction of the electromagnetic navigation procedure.

Additionally, users may visually confirm that the placement of thenavigated medical instrument is positioned in a desired locationrelative to a target tissue within a target area. Additionally, thefluoroscopic-based three dimensional volume can be utilized to visualizethe target area in three dimensions after a procedure is performed. Forexample, the fluoroscopic-based three dimensional volume can be utilizedto visualize the target area after markers are placed within the targetarea, after a biopsy is taken, or after a target is treated.

With particular reference to FIGS. 3A-3B, a method for constructing athree dimensional volume using a plurality of radio-opaque markersplaced proximate the target will now be described and referred to asmethod 300. Method 300 begins at step 301 where a marker placementdevice is navigated to a target area utilizing an electromagneticnavigation system, such as the EMN system 100 (FIG. 1) described above.The navigation of the marker placement device to the target area may beaccomplished using a previously created navigation plan which includesroutes created during the planning phase. In step 303, radio-opaquemarkers are placed within the target area. In one example, fourradio-opaque markers are utilized. However, less than four or more thanfour radio-opaque markers may be used.

In step 305, with the radio-opaque markers placed in the target area,the fluoroscopic imaging device is positioned such that all of theradio-opaque markers placed in step 303 are visible. That is, step 305includes aligning the fluoroscopic imaging device such that it canrotate 30° around the markers with all of the markers visible. In step307, the fluoroscopic imaging device is used to capture a video of abouta 30° rotation of the imaging device 110 about the patient, and thusaround the markers (rotation from −15° to +15°). By rotating up to 15°(on each side) from the centered angle, it can be ensured that themarkers will remain in the images/frames for the entire rotation videoand that the imaging device will not hit the patient or the bed. FIG. 3Billustrates six frames f1-f6 of the captured video. Each of frames f1-f6is an image of the fluoroscopic video showing the different position andorientation of each radio-opaque marker m1-m4 at different points intime of the video, where the fluoroscopic imaging device is positionedat a different angle relative to the patient at each given time.

In step 309, the two-dimensional position of each radio-opaque marker istracked throughout the entire video. In step 311, the marker positionsare constructed in three dimensional using structure-from-motiontechniques and the pose of the fluoroscopic imaging device is obtainedfor each video frame. Structure-from-motion is a method forreconstructing three dimensional positions of points, along withfluoroscopic imaging device locations (camera poses) by tracking thesepoints in a two dimensional continuous video. In step 311, byintroducing markers into the patient, the positions and orientations ofthose markers can be tracked along a continuous fluoroscope rotationvideo, their three dimensional position in space can be reconstructed,and the corresponding fluoroscopic imaging device locations can bedetermined. With the fluoroscopic imaging device locations determinedthrough analysis of the video, the fluoroscopic imaging device locationscan be used to solve for the three dimensional data.

In step 313, a local three dimensional volume is constructed.Specifically, a fast iterative reconstruction algorithm (FIG. 6) is usedto reconstruct a local three dimensional volume at the area of themarkers which correspond to the local anatomy and which soft-tissues arevisible. Step 313 may include reconstructing a global three dimensionalvolume from the acquired two dimensional fluoroscopic data and croppingthe global three dimensional volume at the area of the target to createa “FluoroCT Blob” volume. This cropped volume may be displayed to theuser in the form of raw three dimensional data or as two dimensionalreprojected images. In this cropped volume, all anatomy in the targetarea, including the target tissue, will be visible. The reprojectedimage can be intensified (which does not include the distant denseobscuring objects) by stretching the darker value to be black and thebrighter value to be white to increase differentiation and also sharpenin either three dimension before projection or in the projected images,such that the soft tissue is visually identified.

As described above, the fluoroscopic-based three dimensional volume canbe incorporated into system 100 for multiple purposes. For example, thefluoroscopic-based three dimensional volume can be registered with thepreviously generated three dimensional volumetric data that was utilizedfor navigation of the medical instrument. The system 100 may utilize theregistration between the fluoroscopic-based three dimensional volume andthe previously acquired three dimensional volumetric data to update thecalculated position of the sensor 44 (FIG. 1). Additionally, users mayvisually confirm that the placement of the navigated medical instrumentis positioned in a desired location relative to a target tissue within atarget area.

Turning now to FIG. 4, a method for constructing a three dimensionalvolume using either a single radio-opaque marker placed proximate thetarget or the distal portion of a navigated tool, such as the tip of theextended working channel of the catheter assembly positioned proximatethe target will now be described and referred to as method 400. Althoughdescribed as utilizing the tip of the extended working channel of thecatheter assembly, method 400 may utilize any tool to achieve thisfunction. For example, the tip of a navigated catheter, the tip of abiopsy tool, or the tip of a treatment tool may be utilized. In oneaspect, the tool is navigated transbronchially to the target. In otheraspects, the tool may be of a tool inserted into a patientpercutaneously, for example, a transthoracic navigation of a treatmentdevice such as an ablation device.

Method 400 begins at step 401 where the extended working channel isnavigated to a target area utilizing an electromagnetic navigationsystem, such as the EMN system 100 (FIG. 1) described above. Thenavigation of the EWC to the target area is accomplished using apreviously created navigation plan which includes routes created duringthe planning phase. Method 400 may optionally include the additionalstep of navigating a marker placement device, via the EWC, to the targetarea to place a single radio-opaque marker within the region of thetarget (step 403). In one aspect, step 401 includes percutaneouslyinserting a tool to the target area.

After the EWC or tool is in position, or after the radio-opaque markeris placed, the fluoroscopic imaging device is positioned such that thenavigated tip of the EWC or tool (and/or the placed radio-opaque marker)is visible within the field of view of the fluoroscopic imaging device.That is, step 405 includes aligning the fluoroscopic imaging device suchthat it can rotate 30° around the marker with the marker visible and/oraround the tip of the EWC or tool with the tip of the EWC or toolvisible. In step 407, the fluoroscopic imaging device is used to capturea video of about a 30° rotation of the imaging device 110 about thepatient, and thus around the marker and/or tip of the EWC or tool(rotation from −15° to +15°). By rotating up to 15° (on each side) fromthe centered angle, it can be ensured that the marker and/or tip of theEWC or tool will remain in the images/frames for the entire rotationvideo and that the imaging device will not hit the patient or the bed.Step 407 may include capturing a video of about a 30° rotation aroundthe distal portion of the EWC (and the radio-opaque marker, if placed).If a 30° rotation video is captured, then one angle (in the middle ofthe range) is enough. That is, two projections with 30° between them areenough to confirm or correct three dimensional relation of tools to softtissue.

In step 409, the two-dimensional position of the distal portion of theEWC or tool (and/or the radio-opaque marker, if placed) is trackedthroughout the entire captured video.

In step 411, virtual fluoroscopic images are created from previouslyacquired CT data. The previously acquired CT data is typically the CTdata used during the planning phase to plan a navigation path to thetarget. In step 411, the CT data is manipulated to create a computermodel of fluoroscopic images of the patient. The location of the targetin the virtual fluoroscopic images corresponds to the location of thetarget identified by the clinician during the planning phase. Thevirtual fluoroscopic images generated by the system, based off of thepreviously acquired CT data, depict the field of view that would becaptured by a fluoroscopic imaging device. Additionally, each of thevirtual fluoroscopic images has a virtual fluoroscopic imaging devicepose.

In step 413, each video frame of the fluoroscopic video captured in step407 is registered to the previously acquired CT data by matching each ofthe fluoroscopic video frames to the virtual fluoroscopic images. Instep 415, the fluoroscopic imaging device pose of each video frame ofthe captured fluoroscopic video is determined based on the registrationof step 413. That is, once a fluoroscopic frame is matched to a virtualfluoroscopic image, the virtual fluoroscopic imaging device pose of thevirtual fluoroscopic image can be associated with the correspondingfluoroscopic frame.

In step 417, the origin of the fluoroscopic imaging device posedetermined in step 415 is corrected by using the tracked position of thedistal portion of the EWC or tool (and/or the radio-opaque marker, ifplaced), which is used to compensate for movement of the patient, suchas movement caused by breathing. In step 419, a local three dimensionalvolume is constructed. Specifically, a fast iterative reconstructionalgorithm (FIG. 6) is used to reconstruct a local three dimensionalvolume at the area of the target lesion which corresponds to the localanatomy and which soft-tissues are visible. Step 319 may includereconstructing a global three dimensional volume from the acquired twodimensional fluoroscopic data and cropping the global three dimensionalvolume at the area of the target to create a “FluoroCT Blob” volume.This cropped volume may be displayed to the user in the form of rawthree dimensional data or as two dimensional reprojected images. In thiscropped volume, all anatomy in the target area, including the targettissue, will be visible. The reprojected image can be intensified (whichdoes not include the distant dense obscuring objects) by stretching thedarker value to be black and the brighter value to be white to increasedifferentiation and also sharpen in either three dimension beforeprojection or in the projected images, such that the soft tissue isvisually identified.

Method 400 may also include the additional step (step 421) of completingthe fluoroscopic video captured in step 407 to include virtualfluoroscopic images that are generated by the system, which arerepresentative of fluoroscopic imaging device poses that are outside therange of fluoroscopic imaging device poses captured in the fluoroscopicvideo. Specifically, in aspects, the previously generated CT volumetricdata of the patient which is used to create a navigation plan may alsobe utilized by system 100 to generate virtual fluoroscopic images of thepatient. The generated virtual fluoroscopic images are fluoroscopic-likeimages which display a view to the user of what a fluoroscopic image ofa patient should look like if captured at a given angle by afluoroscopic imaging device. In step 421, the virtual fluoroscopicimages may be used to fill any gaps in the captured fluoroscopic video(captured in step 407). This may include, for example, replacement ofimages, such as frames, of the captured video that are skewed ordamaged. Additionally, or alternatively, this may include, for example,supplementing the captured fluoroscopic video (captured in step 407)with virtual fluoroscopic images that are representative of fluoroscopicimages that are outside the range of angles included in the fluoroscopicvideo. For example, if the fluoroscopic video included a sweep of abouta 30° range about the patient, virtual fluoroscopic images that areoutside the 30° range could be incorporated into the video to generate afluoroscopic video that has a range of greater than 30°.

Method 300 (FIG. 3) and method 400 (FIG. 400) are both used to constructthree dimensional CT volumetric data using fluoroscopic video withoutknowing the fluoroscopic imaging device poses of each of the frames ofthe fluoroscopic video. To this end, each of methods 300 and 400 requiresteps of determining the fluoroscopic imaging device pose of each of theframes of the fluoroscopic video utilizing image-based techniques. Incontrast, and as described in greater detail below, method 500 (FIG. 5)is a method for constructing three dimensional CT volumetric data wherethe fluoroscopic imaging device pose of each of the frames of theacquired fluoroscopic video are determined using a pose/anglemeasurement device, which may include an accelerometer, a gyroscope, ormagnetic field detector to detect the position/pose of the fluoroscopicimaging device relative to the patient.

Method 500 is a method for constructing a three dimensional volume usingeither a single radio-opaque marker placed proximate the target or thetip of the extended working channel of the catheter assembly positionedproximate the target, in conjunction with a fluoroscope anglemeasurement device. Method 500 begins at step 501 where fluoroscopecalibration is performed using a fluoroscope calibration jig to computethe canonical fluoroscope projection parameters and geometry. This isdone once per fluoroscope device, in a setup phase by a technician. Thecalibration jig is used to determine, in an automated process, both theprojection parameters of the fluoroscope (field of view angle) as wellas the geometry of the C-arm: position relative to rotation axis. Theseparameters are sometimes given in technical drawings per fluoroscopedevice, but may also be found using our calibration jig. In step 503,the previously acquired CT volumetric data is imported into the systemalong with the previously generated navigation plan.

In step 505, the extended working channel is navigated to a target areautilizing an electromagnetic navigation technique using a system such asthe EMN system 100 (FIG. 1) described above. The navigation of the EWCto the target area is accomplished using the previously creatednavigation plan which includes routes created during the planning phase.Method 500 may optionally include the additional step (step 507) ofnavigating a marker placement device, via the EWC, to the target area toplace a single radio-opaque marker within the region of the target.

After the EWC or tool is in position, or after the radio-opaque markeris placed, method 500 proceeds to step 509 which includes aligning thefluoroscopic imaging device such that it can rotate 30° around themarker with the marker visible and/or the tip of the EWC or the toolsuch that the tip of the EWC or tool is visible. In step 511, thefluoroscopic imaging device is used to capture a video of about a 30°rotation of the imaging device 110 about the patient, and thus aroundthe marker or tip of the EWC or tool (rotation from −15° to +15°). Byrotating up to 15° (on each side) from the centered angle, it can beensured that the marker or tip of the EWC or tool will remain in theimages/frames for the entire rotation video and that the imaging devicewill not hit the patient or the bed. If a 30° rotation video iscaptured, then one angle (in the middle of the range) is enough. Thatis, two projections with 30° between them are enough to confirm orcorrect three dimensional relation of tools to soft tissue. In step 513,the two-dimensional position and orientation of the distal end of theEWC (and/or the radio-opaque marker, if placed) is tracked throughoutthe entire captured video.

In step 515, the calibration data from step 501 is utilized incombination with measurements from an external angle measurement deviceto compute the fluoroscopic imaging device location (origin-less) inworld coordinates. Specifically, the calibration jig is used, asdescribed above, to automatically find the projection parameters and theC-arm geometry of the specific fluoroscope device by using optimizationmethods. Once these parameters are known, the angle taken from the anglemeasurement device, that is, the angle of the detector of thefluoroscope, determines a unique three dimensional pose for thedetector. It cannot be located in any other place in space other thanthe once explained by the given angle and the setup parameters.

In step 517, a local three dimensional volume is constructed.Specifically, a fast iterative reconstruction algorithm (FIG. 6) is usedto reconstruct a local three dimensional volume at the area of thetarget tissue which corresponds to the local anatomy and whichsoft-tissues are visible. Step 517 may include reconstructing a globalthree dimensional volume from the acquired two dimensional fluoroscopicdata and cropping the global three dimensional volume at the area of thetarget to create a “FluoroCT Blob” volume. This cropped volume may bedisplayed to the user in the form of raw three dimensional data or astwo dimensional reprojected images. In this cropped volume, all anatomyin the target area, including the target tissue, will be visible. Thereprojected image can be intensified (which does not include the distantdense obscuring objects) by stretching the darker value to be black andthe brighter value to be white to increase differentiation and alsosharpen in either three dimension before projection or in the projectedimages, such that the soft tissue is visually identified.

CT reconstruction methods will now be described. CT reconstructionmethods can be divided into Analytic methods (Radon, FDK . . . ) andAlgebraic methods (ART, SART . . . ). Analytic methods assume a veryspecific configuration, such as a full 180° rotation, and reconstructthe CT volume in a single iteration (using some exact formulas).Algebraic methods are more flexible but slower and treat the problem asa large equation system which is solved iteratively (using some gradientdescent method). All methods use projection (three dimensional to twodimensional) and back-projection (two dimensional to three dimensional).

With respect to projection, since each detector pixel is essentially aweighted sum of three dimensional voxels along a ray, the detector imagecan be viewed as a two dimensional projection of the three dimensionalvolume from some fluoroscopic imaging device location. If the threedimensional volume data is already available, then the fluoroscopeimages can be reproduced by projecting it from the known fluoroscopicimaging device locations. A three dimensional reconstruction isconsidered good if its two dimensional projections resemble the observedfluoroscope images it was created from.

With respect to back-projection, at each voxel, the back-projectiondetermines which rays traversed through a particular voxel and sums themtogether. In order to make this determination, the fluoroscopic imagingdevice locations must be known. If the back-projection operator wereapplied to the fluoroscope images of the captured video, then a threedimensional volume could be constructed, but the constructed threedimensional volume would be very blurry and inaccurate because while thetrue center voxel is summed many times, many irrelevant voxelssurrounding the true center voxel are also summed many times. Afterreconstructing a three dimensional volume using one method or another,the quality of the reconstruction is evaluated by taking thereconstructed three dimensional data, projecting it into two dimensionalframes (which may be virtual-fluoroscopic frames) and comparing thosetwo dimensional frames to the original fluoroscopic two dimensionalframes from which the three dimensional data was created. A goal of thevolume reconstruction algorithm is to find three dimensional data whichexplains the two dimensional observations, such that if the threedimensional data were to be projected back into two dimensional frames,those frames would look like the original, real, fluoroscopic frames.When the product is blurry, it means that the projections are blurry anddo not match the real images. In order to address this issue, the methodprovides for a ramp-filter and correction iteration.

Turning now to FIG. 6, a method for reconstructing a local threedimensional volume utilizing a fast iterative algorithm will now bedescribed and referred to as method 600. The three dimensional volumereconstruction algorithm (for example method 600) consists of multipleiterations of projection—back-projection. The goal of the algorithm isto find three dimensional data which explains the two dimensionalobservations of the fluoroscopic imaging device. If it succeeds infinding such data then the three dimensional data found is assumed to bethe three dimensional anatomy of the patient. In each iteration thecurrent three dimensional data found is projected into two dimensions,which should look like the original fluoroscope video, then the twodimensional errors are backprojected back into the three dimensionaldata, and in such a manner the three dimensional data is updated. Thisis repeated several times until the three dimensional data converges andthe process stops. In order to speed up the process, some filtering isdone to the two dimensional projected images before backprojecting themback into three dimensions. The ramp-filter is just an example filterwhich proves to be efficient in speeding up the convergence process.This filter is applied when reconstructing standard CTs with the classicRadon transform. With the classic approach, this process is done in asingle iteration: Filtering (ramp-filter for example), Back-projection.In this method, this is repeated iteratively in several steps.

Method 600 begins in step 601 where the equation begins with an initialvolume V (can just be zeros). In step 603, the volume V is projectedfrom known fluoroscopic imaging device locations into images Q_(i). Forexample, projection may be done, not of the fluoroscopic images, but ofthe fluoroscopic images filtered by a ramp function. The iterations withresiduals, described below, may undergo a ramp-filter beforeback-projections. In step 605, the residuals R_(i)=P_(i)−Q_(i) arecomputed where P_(i) are the observed projections from the capturedfluoroscope video. In step 606, R_(i) is convolved with the ramp-filteras in Radon. In step 607, it is determined whether R_(i) is below apredetermined threshold. If R_(i) is below the predetermined threshold(yes in step 607), then method 600 is complete. If R_(i) is not belowthe predetermined threshold (no in step 607), then method 600 proceedsto step 609. In step 609, R_(i) is back-projected into E, the correctionvolume. In step 611, volume V is set to V+E (V=V+E) and method 600reverts to step 603 where the volume V (now V+E) is projected from knownfluoroscopic imaging device locations into images Q_(i).

Turning now to FIGS. 7 and 8. FIG. 7 illustrates a frame of an originalvideo captured by a fluoroscopic imaging device 700, an image of theframe after being ramp-filtered 703, and an image of the resulting threedimensional volume 705. FIG. 8 is an illustration of a three dimensionalconstruction at angles 801-88, where 801 is 30 degrees, 803 is 60degrees, 805 is 90 degrees, 807 is 120 degrees, 809 is 150 degrees, and811 is 180 degrees.

From the foregoing and with reference to the various figure drawings,those skilled in the art will appreciate that certain modifications canalso be made to the present disclosure without departing from the scopeof the same. For example, although the systems and methods are describedas usable with an EMN system for navigation through a luminal networksuch as the lungs, the systems and methods described herein may beutilized with systems that utilize other navigation and treatmentdevices such as percutaneous devices. Additionally, although theabove-described system and method is described as used within apatient's luminal network, it is appreciated that the above-describedsystems and methods may be utilized in other target regions such as theliver. Further, the above-described systems and methods are also usablefor transthoracic needle aspiration procedures.

Detailed embodiments of the present disclosure are disclosed herein.However, the disclosed embodiments are merely examples of thedisclosure, which may be embodied in various forms and aspects.Therefore, specific structural and functional details disclosed hereinare not to be interpreted as limiting, but merely as a basis for theclaims and as a representative basis for teaching one skilled in the artto variously employ the present disclosure in virtually anyappropriately detailed structure.

As can be appreciated a medical instrument such as a biopsy tool or anenergy device, such as a microwave ablation catheter, that ispositionable through one or more branched luminal networks of a patientto treat tissue may prove useful in the surgical arena and the presentdisclosure is directed to systems and methods that are usable with suchinstruments and tools. Access to luminal networks may be percutaneous orthrough natural orifice using navigation techniques. Additionally,navigation through a luminal network may be accomplished usingimage-guidance. These image-guidance systems may be separate orintegrated with the energy device or a separate access tool and mayinclude MRI, CT, fluoroscopy, ultrasound, electrical impedancetomography, optical, and/or device tracking systems. Methodologies forlocating the access tool include EM, IR, echolocation, optical, andothers. Tracking systems may be integrated to an imaging device, wheretracking is done in virtual space or fused with preoperative or liveimages. In some cases the treatment target may be directly accessed fromwithin the lumen, such as for the treatment of the endobronchial wallfor COPD, Asthma, lung cancer, etc. In other cases, the energy deviceand/or an additional access tool may be required to pierce the lumen andextend into other tissues to reach the target, such as for the treatmentof disease within the parenchyma. Final localization and confirmation ofenergy device or tool placement may be performed with imaging and/ornavigational guidance using a standard fluoroscopic imaging deviceincorporated with methods and systems described above.

While several embodiments of the disclosure have been shown in thedrawings, it is not intended that the disclosure be limited thereto, asit is intended that the disclosure be as broad in scope as the art willallow and that the specification be read likewise. Therefore, the abovedescription should not be construed as limiting, but merely asexemplifications of particular embodiments. Those skilled in the artwill envision other modifications within the scope and spirit of theclaims appended hereto.

What is claimed is:
 1. A system for constructing fluoroscopic-basedthree dimensional volumetric data from two dimensional fluoroscopicimages, comprising: a computing device configured to: create virtualfluoroscopic images of a patient from previously acquired CT volumetricdata, each virtual fluoroscopic image having a virtual fluoroscopicimaging device pose; facilitate navigation of a medical device to atarget area within a patient; receive a sequence of fluoroscopic imagesof the target area from a fluoroscopic imaging device, each frame of thesequence of fluoroscopic images including the target area, a targetwithin the target area, and the medical device positioned relative tothe target; register the virtual fluoroscopic images with the receivedsequence of fluoroscopic images by matching each of the virtualfluoroscopic images with each corresponding frame of the sequence offluoroscopic images; determine a pose of the fluoroscopic imaging devicefor each frame of the sequence of fluoroscopic images based on theregistration of the virtual fluoroscopic images with the sequence offluoroscopic images by associating the virtual fluoroscopic imagingdevice pose with the corresponding frame of the sequence of fluoroscopicimages; and construct fluoroscopic-based three dimensional volumetricdata of the target area in which soft tissue objects are visible basedon the pose of the fluoroscopic imaging device determined for each frameof the sequence of fluoroscopic images.
 2. The system according to claim1, wherein the computing device is further configured to: detect missingframes from the sequence of fluoroscopic images; and supplement themissing frames with corresponding virtual fluoroscopic images.
 3. Thesystem according to claim 2, wherein the fluoroscopic-based threedimensional volumetric data is constructed based on the sequence offluoroscopic images and the corresponding virtual fluoroscopic images.4. The system according to claim 1, wherein the computing device isfurther configured to track a two dimensional position of the medicaldevice navigated to the target area in each frame of the sequence offluoroscopic images.
 5. The system according to claim 1, wherein thecomputing device is further configured to reconstruct positions of themedical device throughout the sequence of fluoroscopic images using astructure-from-motion technique and the pose of the fluoroscopic imagingdevice for each frame of the sequence of fluoroscopic images isdetermined also based on the positions reconstructed.
 6. The systemaccording to claim 1, wherein the pose of the fluoroscopic imagingdevice for each frame of the sequence of fluoroscopic images isdetermined also based on an external angle measuring device coupled tothe fluoroscopic imaging device.
 7. The system according to claim 1,wherein the computing device is further configured to: synchronize theframes of the target area; and compensate for shifts in the fluoroscopicimaging device or patient movement to correct construction of thefluoroscopic-based three dimensional volumetric data.
 8. The systemaccording to claim 1, wherein the medical device is at least one of acatheter assembly including an extended working channel configured to bepositioned within a luminal network of the patient, a radio-opaquemarker configured to be placed within the target area, wherein theradio-opaque marker is at least partially visible in the sequence offluoroscopic images, or a tool configured to percutaneously access thetarget area.
 9. The system according to claim 1, wherein the computingdevice is further configured to: crop a region of interest from thefluoroscopic-based three dimensional volumetric data; project the regionof interest cropped from the fluoroscopic-based three dimensionalvolumetric data onto the frames; and sharpen or intensify at least oneof the region of interest or the frame to identify soft tissue objects.10. The system according to claim 1, wherein the sequence offluoroscopic images of the target area includes at least about a thirtydegree rotation about the target area.
 11. A method for constructingfluoroscopic-based three dimensional volumetric data from twodimensional fluoroscopic images, comprising: creating virtualfluoroscopic images of a patient from previously acquired CT volumetricdata, each virtual fluoroscopic image having a virtual fluoroscopicimaging device pose; receiving a sequence of fluoroscopic images of atarget area from a fluoroscopic imaging device, each frame of thesequence of fluoroscopic images including the target area, a targetwithin the target area, and a medical device positioned relative to thetarget; registering the virtual fluoroscopic images with the receivedsequence of fluoroscopic images by matching each of the virtualfluoroscopic images with each corresponding frame of the sequence offluoroscopic images; determining a pose of the fluoroscopic imagingdevice for each frame of the sequence of fluoroscopic images based onthe registration of the virtual fluoroscopic images with the sequence offluoroscopic images by associating the virtual fluoroscopic imagingdevice pose with the corresponding frame of the sequence of fluoroscopicimages; and constructing fluoroscopic-based three dimensional volumetricdata of the target area in which soft tissue objects are visible basedon the pose of the fluoroscopic imaging device determined for each frameof the sequence of fluoroscopic images.
 12. The method according toclaim 11, further comprising: detecting missing frames from the sequenceof fluoroscopic images; and supplementing the missing frames detectedwith corresponding virtual fluoroscopic images.
 13. The method accordingto claim 11, further comprising tracking a two dimensional position ofthe medical device navigated to the target area in each frame throughoutthe sequence of fluoroscopic images.
 14. The method according to claim11, further comprising reconstructing positions of the medical devicethroughout the sequence of fluoroscopic images using astructure-from-motion technique, wherein determining the pose of thefluoroscopic imaging device for each frame of the sequence offluoroscopic images is also based on the positions reconstructed. 15.The method according to claim 11, wherein determining the pose of thefluoroscopic imaging device for each frame of the sequence offluoroscopic images is also based on an external angle measuring devicecoupled to the fluoroscopic imaging device.
 16. The method according toclaim 11, further comprising: synchronizing the frames of the targetarea; and compensating for shifts in the fluoroscopic imaging device orpatient movement to correct construction of the fluoroscopic-based threedimensional volumetric data.
 17. The method according to claim 11,wherein navigating the medical device to the target area within thepatient includes navigating at least one of an extended working channelof a catheter assembly to the target area, a radio-opaque marker to thetarget area, wherein the radio-opaque marker is at least partiallyvisible in the sequence of fluoroscopic images, or a tool configured topercutaneously access the target area.
 18. The method according to claim11, further comprising: cropping a region of interest from thefluoroscopic-based three dimensional volumetric data; projecting theregion of interest cropped from the fluoroscopic-based three dimensionalvolumetric data onto the frames; and sharpening or intensifying at leastone of the region of interest or the frame to identify soft tissueobjects.